In a System Area Network (SAN), such as an InfiniBand (IB) network, the hardware provides a message passing mechanism that can be used for Input/Output devices (I/O) and interprocess communications (IPC) between general computing nodes. Processes executing on devices access SAN message passing hardware by posting send/receive messages to send/receive work queues on a SAN channel adapter (CA). These processes also are referred to as ‘consumers.’
The send/receive work queues (WQ) are assigned to a consumer as a queue pair (QP). The messages can be sent over five different transport types: Reliable Connected (RC), Reliable Datagram (RD), Unreliable Connected (UC), Unreliable Datagram (UD), and Raw Datagram (RawD). Consumers retrieve the results of these messages from a completion queue (CQ) through SAN send and receive work completion (WC) queues. The source channel adapter takes care of segmenting outbound messages and sending them to the destination. The destination channel adapter takes care of reassembling inbound messages and placing them in the memory space designated by the destination's consumer.
Two channel adapter types are present in nodes of the SAN fabric, a host channel adapter (HCA) and a target channel adapter (TCA). The host channel adapter is used by general purpose computing nodes to access the SAN fabric. Consumers use SAN verbs to access host channel adapter functions. The software that interprets verbs and directly accesses the channel adapter is known as the channel interface (CI).
Target channel adapters (TCA) are used by nodes that are the subject of messages sent from host channel adapters. The target channel adapters serve a similar function as that of the host channel adapters in providing the target node an access point to the SAN fabric.
Channel adapters may be either directly connected together or can be connected through the SAN fabric. The SAN fabric comprises one or more switches and/or one or more routers. In any case, the links between the channel adapter ports, switch ports, and router ports are point to point and have separate conductors in each direction. Each of the two ports connected by a link has multiple virtual lanes (VL). Virtual lanes provide a mechanism for creating multiple virtual links on a single physical link. Associated with each virtual lane is a virtual lane buffer which is used to control the flow of packets. The flow control is credit based which means that the transmitter sends packets only if there is sufficient empty space in the receiving virtual lane buffer.
Some of the virtual lane buffer space at the receiver accommodates the round trip delay on the link. As links get longer, the round trip delay increases and larger virtual lane buffers are required to keep the link going at full speed. With a given size of virtual lane receiver buffers, at some distance the round trip link delay is enough that the cerdits do not arrive at the transmitter quickly enough to keep the link going at full speed.
Implementations of channel adapters, switches, and routers limit the virtual lane buffering to the amount anticipated by the configuration, and most configurations limit the links to less than 300 meters. What is needed is a way to extend links to many kilometers while maintaining the full link speed without adding large virtual lane buffers to every channel adapter, switch, and router port. By introducing a repeater that has large virtual link buffers, very long distances can be achieved. At some point, the format of the flow control packets must be expanded to accommodate the larger buffers. The repeater may or may not be visible to the various layers of link management software and firmware, and making the repeater invisible requires further changes to the flow control process.
The repeater function could either be incorporated into switch or router ports or it could be a stand alone device in which case it looks like a two port switch. Additionally, the repeater function could be constructed to be transparent to management software and firmware.